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ABSTRACT 



This paper addresses the emerging evaluation and controlling 
system of Germany's Bundesanstalt fuer Arbeit, which is responsible for 
employment promotion . Part 1 discusses changes in employment administration 
in European countries, a process that has occurred in the Federal Republic of 
Germany, although most of the basic structures have been maintained there. 
Part 2 addresses recent innovations in employment promotion, including 
placement and guidance, from the perspective of control and evaluation, 
mainly in the context of the new law. Social Code III. It discusses Social 
Code III, which, since 1998, defines a framework that includes new, closely 
interrelated provisions for controlling and evaluation and that can be 
described by the key words "new promotion philosophy," "new programs," 
"decentralization," and "regionalization." Part 3 focuses on output 
evaluations now prescribed by law, as well as the concomitant concept of 
competitive performance, which opens up new opportunities. Part 4 discusses a 
system of target agreements that specify labor market-related impact targets. 
A second focus is quality management. After examining these specific aspects, 
part 5 offers a systematic framework showing the role played by monitoring, 
control, and evaluation in relation to current issues. Part 6 looks at 
parallel developments at the European Union level. The report contains 29 
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Abstract 



In many European countries, employment administrations have undergone changes in re- 
cent years in order to become more effective and efficient. Although the same process 
occurred in the Federal Republic of Germany (FRG), most of the basic structures have 
been maintained. At the federal level employment promotion has been subject to the pro- 
visions of Social Code 111 since eaiiy 1998, this law gave new impetus to this trend and 
defined a framework characterized by the key words; new promotion 'philosophy*, new 
schemes, decentralisation, and regionalisation. It includes new provisions for controlling 
and evaluation. 

Major elements of this new system will be discussed below: particularly important are the 
statutoiy Lingliedcrifngshjlanzen (output evaluations) and the goals used as a means of 
control of the impact on the labour-market. 

Lingliederiingshilanzcn include detailed and standardised input and output indicators, 
but no outcome indicators. The most important part of the Eingliederiingshilanz are the 
data on the employment status ol each individual “a reasonable period after completion 
of a programme”. Such evaluations are a step in the right direction; however, the very 
specific provisions of the law will also generate problems. Clearly, they will initially only 
include “gross” results on short-term re-employment successes, and ignore other desi- 
rable objectives, in particular qualitative aspects, such as “employment in keeping with 
qualifications”. 

To control placement, counselling and deployment of labour-market policy instruments, 
labour market-related policy priorities ai e agreed at. all levels of the employment admini- 
stration. However, currently this goal controlling is only intended to affect factors that 
can be quantified in the short term - a fundamental problem. For goals affecting the la- 
bour-market, quality management will be needed which must then be integrated into the 
emerging system of goal agreements. 

One section of the paper outlines a systematic framework that helps to put the individual 
aspects described in perspective. The relationships between the concepts of monitoring, 
controlling and evaluation are established and the current problems are discussed. Final- 
ly, parallel developments at the level of the EU are mentioned. 



1 IntrodiictioM 

In many European countries, employment administrations have undergone changes in re- 
cent years in order to become more effective and efficient. Although the same process 
occurred in the Federal Republic of Germany (FRG), most of the basic structures have 
beeii maintained. 

In Germany, the Bundesanstalt fiir /-Xrbeil (BA) is responsible for employment promo- 
tion. The Bundesanstalt has always been an independent institution with an autonomous 
administration in which employers, trade unions and government (the appropriate federal 
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ministries, the federal Laender and the municipalities) are all equally represented. It is the 
.Federal Ministry of Labour's responsibility to ensure that the Bundesanstalt acts within 
the law, but it may not instruct the BA how to act. 

'The three-tiered organisation of the BA reflects the federal structure of the FRG: 

- the head office (federal level) is in Nuremberg; 

- there are 10 regional employment ofTice districts (Landesarbeitsamtsbezirke) and 

- 181 local employment otlices with 640 bianch offices. 

At each of these levels, an autonomous administrative body is responsible for decision- 
making. At the federal level, the decision-making body is a quasi-legislative body: the 
Board of Governors. This Board may direct action on matters not determined by the law 
provided for by Social Code III. This law- has governed employment promotion nation- 
wide since 1 January, 1998. 

In the FRG, in contrast to most other EU member states, employment promotion is 
mainly financed from contributions to social security funded by employees and employers 
in equal shares. At present, a total of G .'iyo of gross income is deducted at source and 
allocated to social security. (The federal government may lower this rate once the labour 
market situation improves.) If the Bundesanstalt runs a deficit, the federal government 
must approve its budget: this clearly limits its autonomy. 

Employment offices will pay unemployment benefits to unemployed persons for a limited 
period depending on age, e.g. for younger unemployed persons, the maximum duration is 
one year. Once they are no longer entitled to receive such benefits, destitute unemployed 
persons receive unemployment relief from federal funds. If they are destitute but not en- 
titled to insurance benefits, they receive welfare benefits from local governments. All in 
all, about 20% of people out of work do not receive any payment from the employment 
office; about 10% to 15% depend on welfare payments they receive from the municipali- 
ties. 

It is the municipalities that should fund measures for vocational integration of these un- 
employed persons. They co-operate with the employment offices which continue to bear 
the primary responsibility for unemployment benefits, job placement, counselling, voca- 
tional guidance and measures (“one-stop-shopping”). For some years now, employment 
offices have no longer been solely responsible for Job placement and counselling: today 
there are also private employment agencies and vocational guidance agencies that cater 
specifically to the needs of young people or adults. 

In the FRG, just as in other countries, the main pillars of labour market policy are: 
training and qualification measures; 

job, creation measures (must be work that would not otherwise be done and which is 
in the public interest); 

wage subsidies for hard-to-place unemployed people; 
start-up subsidies for self-employed and other ventures. 
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After German unification and the ensuing transformation in eastern Germany, labour 
market policy efforts were stepped up considerably. In 1998, the .Bundesanstalt's budget 
was DM 100 billion (about Euro 50 billion), quite a respectable sum, about 10% of 
which was covered by a grant from the federal government. 

As the labour administration increased its activities, there were calls for organisational 
change because many still considered the BA to be over-centralised and bureaucratic. At 
the same time, some felt that there should be more thorough evaluation and “control- 
ling” of the BA’s work, in the sense of greater oversight, and specifically, greater moni- 
toring and feedback to decision-makers at all levels of the organisation. 

These changes are reflected in the new law. Social Code 111. Developments in this area 
will be described below. 

This paper addresses the BA's emerging evaluation and controlling system. It should be 
noted that this discussion of evaluation and controlling primarily relaterS to the interface 
with employment research. Many other aspects which might otherwise be considered to 
fall within the scope of a discussion of organisational controlling will not be included in 
this paper. 

.However, the issue of controlling is increasingly affecting employment research in the 
following ways: 

more questions are being asked about output and outcome, about the impact of the 
BA's activities on the labour market, i.e. of guidance, placement and programmes; 

secondly, the evaluation of labour-market policy measures is directly affected - this 
is a field of research the Bundesanstalt fur Arbeit Institut fur Arbeitsmarkt- und 
Berufsforschung (lAB - Institute for Employment Research) has been focussing on in 
recent years. This work not only integrates the results of impact research into the 
BA's emerging controlling system, but also 

• places new demands on research; 

• makes new types of data available; 

• is paving the way for new forms of co-operation between academic research and 
practical work within the BA. 

The new legal situation created by Social Code III, Section 280, is instrumental in this 
respect, becau.se it says that the BA must not only examine and evaluate the labour mar- 
ket situation and its trends, but must also track them. 

“... the impact of active employment promotion by 

1. compiling statistics, 

2. engaging in labour market and occupational research, and 

3. reporting thereon.” 
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The wording of the law makes clear that in this context, outcome control and evaluation 
are not solely a task of the lAB or of research, but of the BA as a whole at all levels. A 
wide range of well co-ordinated activities is called for. 

.Part 2 of this paper will address recent innovations in employment promotion, including 
placement and guidance, from the perspective of control and evaluation, mainly (but not 
exclusively) in the context of Social Code 111. The key word here is: Jeceniralisalion. 

The paper discusses output evaluations {Eifi^'liedcriingshi/anz) now prescribed by law, 
as well as the concomitant concept of competitive performance, which, while having 
their limits and associated risks, open up new opportunities (Part 3). The BA has de- 
signed a system of target agreements which specify labour market related impact targets. 
This system, together with aspects of quality management, also deserve the reader’s at- 
tention (Part 4). 

After examining these specific aspects, the paper offers a systematic framework showing 
the role played by monitoring, control and evaluation in relation to current issues (Part 
5). This is followed by a look at parallel developments at European Union level (Part 6). 

The author does not wish to contribute to the general discussion about merits and de- 
merits of applying private enterprise controlling activities to public administrations. The 
following ideal-typical differences would make this difficult (Postlep, 1994); 

in the one case, goods are made available to be sold in the private economy; in the 
other, the goods are (almost exclusively)' public; 

in the one case, purchasing power driven demand and profitability are the criteria, in 
the other, the objectives are more complex, partially inconsistent; there are no clearly 
defined and prioritised administrative policy decisions with the vested interests of the 
parties at stake; 

and there are traditional problems created by administrative acts, like biased spending 
behaviour and lack of output planning. 

This generalised description may help to explain a number of specific problems that sur- 
face downstream and which will also be addressed below. 



2 Iiiiiovatioiis in (lie Held of einployiiieiit promotion 
2.1 Ai beitsamt 2000 

Let us take a closer look at some of the changes in the employment administration and 
the reasons for these changes. Although the ArheiLsaml 2000 (Employment Office 2000) 
concept and related developments are of major importance, this paper will not dwell on 



' Plnccmcni and counselling can be ori'ciccl by privalc cnicrpriscs for rcnuincialion. Public aclininislra- 
lions may pioclucc goods lhal can be sold. Ncvcrilicicss Ihc idcal-lypical disiinclion made here illii- 
slrales ihe iransfer of the concepi of conirolling conlemplaled herein lo publiciadminislralions and the 
relaled special dirfieiillics. 
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internal organisation and organisational controlling changes (Bundesanstalt fur Arbeit, 
1996b). Figure 1 shows the areas of activities as they look now, subsequent to the rede- 
finition of organisational objectives. This new model has been tested in a number of pilot 
employment offices and the initial results are in (Kulozik, 1998). The introduction of 
team-based organisation and flat hierarchies will doubtlessly cause a certain amount of 
friction and rnust be seen as a long-term effort. The process is supported by other per- 
formance-based management trials in some employment offices. The Bertelsmann Foun- 
dation is involved in this endeavour and has contributed valuable experience from similar 
reengineering efforts in local governments (Bundesanstalt fur Arbeit 1995 and 1996).^ 

2.2 Social Code ni: Decentralisation and regionalisation 

Social Code 111 gave new impetus to the “revamping” of the employment administration. 
It defined a framework which can be described with the key words: new promotion 'phi- 
losophy', new programmes, decentralisation, and regionalisation. It includes new, closely 
interrelated provisions for controlling and evaluation. 

At this point, we are not so much interested in the “new promotion philosophy”, which 
is reflected in the new instruments (Ammermuller, 1997; Brinkmann/Kress, 1997) to a 
certain extent only. The objective of directly balancing the labour market is stressed more 
than before, i.e. specific problem groups on the labour market should be specifically tar- 
geted (Section 7 (3)); the target group is further limited to persons paying social security 
contributions and persons receiving payments to replace earnings under Social Code HI. 
At the same time, it is stressed that local governments have responsibility for employ- 
ment policy (although the nature of this is ambivalent) - this point is addressed' later. 

Nor is it the goal of the author to look at the new instruments in detail in this paper, al- 
though there might be a need to assess new approaches such as special integration con- 
tracts in particular, or subsidies to social plans easing the effects of mass layoffs." 

Special mention must, however, be made of “uncommitted funds for labour promotion” 
provided for under Social Code 111, Section 10; according to this provision, employment 
offices have discretionary powers to earmark up to 10% of their funds for active em- 
ployment promotion. They can take advantage of this legal provision to expand employ- 
ment promotion options. This amounts to funds in the order of a maximum of DM 2 to 3 
billion.”' Consequently, each individual employment office has considerable flexibility to 
act with a local focus, although they must obsei^ve the same objectives and principles as 
apply to statutoiy support and they may not use the funds simply to top up statutory 
payments. 




■ See Seciion 4.2 

■’ In 1998, the lAB laiinclied two long-term representative projects to e\'alii:ite the wide range of com- 
pany-related hiring incentives (lAB projects 10-497 :ind 10-498). Regarding socitil plans, the lAB 
conimissioned the Institiit Arbeit nnd Tcchnik in Gelsenkirchen to engage in parallel research (lAB 
project 10-508 A). 

" In their 1998 budget planning, the einploytnent ofriccs cstiinaled that they would receive about DM 
700 million in uncommitted funding, i.e. ;ibout 2.7% of the btidget item for re-cmploymenl. 
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These uncommitted funds should open up the way for new and alternative support mea- 
sures, as was the case with the new types of co-fmanced public employment called 
“structural adaptation schemes” - a special job creation scheme supporting structural 
policy. These may be “model projects”, but they must also be systematically monitored 
and evaluated with a view to promoting the interregional learning process (Brinkmann 
1998b, Brinkmann/Schmitt 1999). 

Uncommitted funds arc already an important resource for the more comprehensive re- 
gional authority of local employment offices as provided for by Social Code III. Before 
proceeding, it might be useful to clarify certain terms. Strictly speaking, regionalisation 
means that the regional levels are empowered in the decision-making process, while de- 
centralisation means that the implementation of defined programmes is delegated to a 
lower regional level (Hild 1997, 224). In this sense, the new law decentralised a large 
part of employment promotion away from the federal level, although decision-making 
has been shifted as well. Clearly, the main intention of this was to increase the employ- 
ment administration’s effectiveness and efficiency (Brinkmann 1998a). 

Pursuant to Section 9, employment promotion shall be provided for mainly by local em- 
ployment offices acting in cooperation with all parties active on the local labour market. 

This might not sound very new compared to the former Employment Promotion Act. 
However, there is a new wrinkle in that amendments to other parts of the Social Code 
(Social Code IV, Section 71b) provide that all discretionary spending to promote em- 
ployment must be concentrated in one single budget item, the Eiiigliederimgslilel, and 
that all funds allocated to this item must be assigned to the employment offices (with 
minor exceptions) to be accountable for and to manage^ The employment offices must 
make funds available for each type of discretionary scheme, taking into account the si- 
tuation and trends on the local labour market. What is totally new is that employment of- 
fices are now free to decide independently on their priorities, on how much weight they 
want to assign to e.g. training programmes, job creation schemes or hiring subsidies to 
companies in their region. 

Any evaluation of efficiency and effectiveness of labour market policy must therefore gi- 
ve more consideration than before to regional flexibility and take into account the various 
support strategies developed by the employment offices. This and other legislative chan- 
ges (such as permitting the involvement of third parties in placement efforts) have provi- 
ded the employment offices with new authority and possibilities for cooperation; their 
ability to make good use of these is encouraged by the internal administrative reforms 
mentioned above. 




■ Active labour market policy runding is allocalcd ccnlrally and distributed lo Ibe regions by decision of 
the tripartite (representatives of employers, employees, and public institutions) autonomous admini- 
strative bodies of the Bundesanstalt fiir Arbdit, and subject to approval by the Federal Minister of La- 
bour. Most of the funding comes from contributions p.iid to the Bundesanstalt fiir Arbeit and is subsi- 
dised from the fedenil budget. Within the scope of the funds alloctited to them, the employment offi- 
ces are now free lo decide on how they wish to use the funds assigned to them and on the mix of in-' 
struments they want lo implement in their particular region. 
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Certainly, regionalisation could be interpreted to be much more extensive than is at pre- 
sent the case, mainly on the basis of Social Code 111. There were, and still are, concepts 
that are much broader whose goal is to integrate labour market policy into a comprehen- 
sive regional employment policy (Kress, 1996). Be that as it may, after these changes, la- 
bour market policy in Germany looks rather different from the way it used to be. 

2.3 Cooperation with third paHics 

It should not be forgotten that other developments in the context of labour market policy 
have strengthened and assigned more weight to the regional levels. Firstly, the law on 
welfare payments provides for increased stronger efforts by local governments to get un- 
employed welfare recipients back to work. This must be seen against the background of 
a continuously growing number of unemployed persons without any entitlement to bene- 
fits either under the Employment Promotion Law or Social Code III, or whose benefits 
are less than the welfare payments 

Secondly, the Bundesanstalt had lost its monopoly on placement and guidance to some 
extent even prior to the introduction of Social Code Til. As a result, employment offices 
were obliged to rethink their counselling and placement activities. A phased method has 
been devised whereby a stronger emphasis is placed on the job-seekers’ own initiative 
and responsibility, as Social Code 111 stresses in various provisions (Sections 2 and 19). 

What is more important in the context under discussion here is that Social Code III, Sec- 
tion 37, permits employment offices to involve third parties in their placement efforts, if 
the person seeking training or a job consents. Such third parties might be Chambers of 
Commerce or Trade, associations, welfare offices or private agencies. Placement agen- 
cies may be established (e.g. jointly by the employment and the welfare offices) or exi- 
sting ones may be approached. This means that cases involving persons with special pla- 
cement handicaps that require intensive counselling and placement efforts, for which 
neither the employment nor the welfare offices have the necessary human resources, may 
be outsourced to third parties who might work on commission to try to find a placement 
according to the “Maatwerk method” developed in the Netherlands. 

As third parties in the form of placement agencies gain profile, it would seem wise to re- 
cord and systematically evaluate such local governments outsourcing. Only such evalua- 
tion will show the (net) results, which can then be weighed against the sometimes consi- 
derable (additional) expenses.^’ 



3 Social Code in : output evaluations 




In very general terms, this means: administrative decentralisation, regional employment 
offices with budget responsibility for re-employment activities, and greater co-operation 



6 



Finally, all this entails a comparison ofcosis and bciiefits of “regular;’ placement work by employment 
offices. For all newly piloted promotion programmes (including placement promotion activities) paid 
for with nncommitlcd funds now provided for under Social Code 111, Section 10, the Bundesanstalt 
budget includes funding to rmance a limited amount of resctirch which may be requested by the regi- 
ons. 



lAB Liibour Market Resctirch Topics 3G (1999) 



13 



9 



to make labour market policy efficient and responsive to regional needs. All of this pre- 
supposes transparency so that progress can be tracked easily and undesirable develop- 
ments identified. It is precisely for this reason that employment offices are now required 
to draw up output evaluations {Eingliedenmgshilonzeu) that also include input indica- 
tors to reflect active employment promotion activities that they have funded. 

3.1 Purpose and design 

As from mid-1999 (for fiscal year 1998), all employment offices will be required to com- 
pile and publish annual output evaluations: 

According to article 1 1 of the law, which specifies in some detail which data must be in- 
cluded, evaluation reports must contain the following information: 

volume of funds and distribution, 
average per capita expenditure on measures, 

number and distribution of different activities and inclusion of groups requiring spe- 
cial assistance, 

proportion of women involved, 
placement rate, 

rate of participants in programmes in employment/remaining unemployed, 
general conditions characterising the regional labour market, 
longitudinal development of programmes 

The most important part of the Eiiigliedeningsbilanz is the data on the employment 
status of each individual “a reasonable period after completion of a measure”. This 
means that each employment office is required to categorise the various measures offered 
and show how many of the participants are registered as unemployed six months after 
each measure has ended. This short-term perspective (over six rhonths) is insufficient to 
indicate how many are employed, or employed in the open labour market, as had been 
planned originally. Where possible, the report must include data on groups of people 
who need special help. Because of the nature of the statistics that are currently available 
and to keep the data manageable, this will initially be limited to the following groups: 
long-term unemployed, disabled/groups classified as disabled, older workers (50+) and 
persons returning to working life. Women are listed separately. . 

The Bundesanstalt's head office has been asked to establish uniform evaluation guidelines 
in order to ensure comparability of regional and national output evaluations. Figure 2 is 
an excerpt from the mode Eingliedenmgshilanz showing the ratio of subsequently un- 
employed participants to all participants. The first evaluation report in mid-99 will al- 
ready be able to include these ratios for the 1998 participants in most of the (broadly de- 
fined) programme categories listed. 

The German Parliament explained the reasons for these reports as follows: 

“The provision represents an additional element of control where budgetary responsibil- 
ity is largely decentralised. Although decentralisation vastly increases the opportunity for 
effective and efficient employment promotion, it also increases the risk of mismanage- 
ment and biased decision-making. What is required is a method of control which does 
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Excerpt from model 'Eingliedeningshilanz' according to section 11 Social Code III with follow-up information, indicators referring to 
proportion of participants not unemployed half a year after completion of measure (according to computerised register) 
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1) According to section 7 (3) SC III. It is planned to include further groups in the next stages of the EingUedenmgsbilam. 

2) Total of hard-to-place unemployed, e. number of persons showing at least one of the characteristics 3-6. 



not in turn undo the advantages of decentralisation ... Although individual employment 
office districts are coping with different circumstances, the evaluation reports will allow 
for a degree of performance comparability. This in turn will induce the employment of- 
fices to compete for the best performance, something which is currently not possible, be- 
cause the absence of transparency makes it impossible to see how well any one office is 
doing.” (Deutscher Bundestag 1996, p. 155), 

Eingliedenmgshilanzen are therefore considered to be an important instrument that both 
clearly show the “effectiveness of promotion measures” and stimulate the employment 
offices to deploy measures more effectively and more efficiently as a result of managed 
competition. Only one phrase in the law betrays any doubts in the validity and usefulness 
of this procedure. The text says that in addition to these output evaluations, which con- 
sist of largely predefined indicators, there should also be data that “provide further in- 
formation about the activities and their impact on the local labour market...”. (Section 
1 1. paragraph 3). 

To what extent can such output evaluations actually be valid? What problems do they 
create, and are there solutions to these problems or not? 

3.2 General critique 

Output evaluations are a step in the right direction: for the first time, standardised data in 
the form of “gross results” similar to process statistics on the employment status of par- 
ticipants in active policy measures will be available for all major fields of labour market 
policy on a regular basis. Until now, such basic data were only available for isolated 
cases, covered longer intervals in the case of various instruments and usually required 
special complex studies. 

“Gross results” means: data on the employment status of participants which cannot be ^ 
attributed exclusively to a specific measure. This effect can only be shown once it is 
known how many participants would have found work even without any assistance .- 
' one of the most complex criteria for evaluation. Only affer these persons have been “de- 
ducted” can “net” results be obtained (i.e. such for which causal analysis can attribute 
the resulting employment status to participation in programmes)’. This essential differ- 
ence is frequently ignored. 

Notwithstanding both the law-makers’ extremely ambitious goals for the Ein- 
gliedcrungshilanz and the possible and probable misinterpretation (gross = net), this type 
of evaluation is to be welcomed. It will be of critical importance as part of a monitoring 
system that must of necessity become ever more detailed and complex. There is one 
other limitation that should be mentioned, however: its timing. It will be submitted too 
late to be used for direct controlling purposes, i.e. to influence the deployment of meas- 
ures in the course of the same fiscal year. 
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Anollier factor (o be obscr\'cd is tlic time clcmcnl. Tlic measure is intended lo get people back in work 
as quickly as possible, i.e. more quickly Ilian had they received no assislancc. The difference between 
when an individual becomes re-cmplo.\'cd with and without assislancc shows Ihe measure's net effect. 
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Output evaluations with such detailed legal requirements will cause other problems as 
well. Initially,, they can obviously only include data on short-term re-employment suc- 
cesses. It is clear that some measures (e.g. preparatoi^y measures) can only indirectly lead 
to re-employment. Similarly, long-term re-employment and qualitative career changes 
that come out of the measures in question are not reflected in the results. Another prob- 
lem is that some of the measures have other purposes as well (e.g. vocational stability 
and better, employability, structural improvement of the labour market, etc.) and other 
(incidental) benefits might play a role**. 

Different labour market structures may have a much greater impact on re-employment 
than anything an individual employment office can do in the way of promotion. This 
means that whether a participant in a job creation or skill improvement measure will be- 
come re-employed or not will partly depend on the conditions on the local labour market. 
The general conditions characterising the regional labour market were only included in 
the Einglicdeningshilanz as an after-thought, and then in such a brief and concise form 
that they cannot do more than hint at any real problems. In addition, comparisons over 
time are often more informative than the cross-sectional type of comparison provided for 
initially. Yet, here again, the data on the development of the measures over time (which 
was also added later, although this was not the original intention) is so thin, that it can do 
little more than intimate that there is a problem. 

All of the studies on successful re-employment completed so far show that any more in- 
tensive concentration on target groups has a negative result on their re-employment rate 
(Blaschke/Nagel, 1995). Evaluations abbreviated to short-term output evaluations will 
probably trigger competition for the highest re-employment rates to the detriment of 
those persons who are most difficult to place. The dilemma is highlighted by the fact that 
the “groups requiring most help” are certainly high on the agenda, but this does not in 
itself provide the solution. Each of these categories of statistics can hide very different 
types of problems and impedimdnts to employability. This factor should.be carefully ob- 
served when referring to these statistics. 

Figure 3 summarises the general problems potentially arising from the output evaluations 
generated in the form required. It will be important to identify any unwanted effects of 
this type of output evaluations and to act to contain them. A causal analysis of certain 
effects will be required to avoid misinterpretations of these evaluations. 

3.3 The monitoring system: n work in progress 

It is important to note that EingliedenmgshHanzcn will not be submitted until after the 
fiscal year has ended. For this reason it was suggested that interim results be compiled 
and made available on an on-going basis. These interim results may be used to determine 
corrective action during the course of a fiscal year. 

The compilation of output evaluation reports means that different files of the BA will 
have to be integrated. This will lead to a single personal client identification number that 
shows all of the employment offices' activities (benefits, counselling, placement. 



** See Bach/Spilznagel, (1992), Aiilorcngcincinscliafl (1997), and Aiilorengemeinschan ('1998). 
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Figure 3 



Mnior criticism of the new 'EimiHederumishilanzen' 



Only short-tem monitoring 
(unemployed after six months yes/no) 

No qualitative goals 
(employability, working conditions) 

No outcome, indicators for the aggregate 
labour market level 

(easing the burden^, structural improve- 
ment) 

Monitoring at the level of 'gross' results, 
which may not be interpreted as net ef- 
fects of measures 

Only partial controlling for creaming 
effects 

Emphasis on cross-sectional regional 
comparison may lead to the wrong con- 
clusions, as regions may only be partially 
comparable for structural reasons 




“There will be an additional part showing the iininediate quantitative effeets of the measures on the lo- 
cal labour market (at the 'gross level'). 



lAB Labour Market Research Topics 36 (1999) 



14 



implementation of measures) and allows a data match indicating whether an individual is 
employed or unemployed - an objective which goes far beyond short-term monitoring. 

3.4 Significance for research 

The introduction of the output evaluations described in this paper will mark the begin- 
ning of a new era for employment promotion evaluation instruments. The routine avail- 
ability of gross data on the short-term employment status of participants in employment 
measures, together with the expansion and merging of data required to obtain this infor- 
mation, will close significant information gaps. This will open up new options for apply- 
ing BA statistics and for control; at the same time, new demands will be made on evalua- 
tion research, and new data will be available for special studies primarily in the form of 
causal analyses. 

.Future records will be able to include a new type of detailed stmctural information on 
participants in measures; they will enable users to match long-term data (e.g. with rec- 
ords on employees in Jobs requiring payment of social-security contributions); it will be 
possible to take samples from the newly-created IT systems (e.g. of surveys of measure 
participants, etc.). The lAB's outcome research will be able to use this new source of 
data - in fact, it has already taken action"’. 

4 Outcome goals for the labour market 

4.1 Target agreements 

Eingliedenmgshilanzen in their present form will be essential for the BA's controlling 
system, but they will be insufficient by themselves. They include the input and output in- 
dicators (expenditures, participant's subsequent employment status); however, the infor- 
mation they provide about their impact on the labour market is at best indirect and in- 
cluded only in an additional table (showing how much pressure programmes really did 
take off the regional labour market) and in some ancillai^ comments - apart from this, 
they do not include any “outcome” indicators. 

There is a second series of objectives that was already touched upon in relation to the 
output evaluations and which has also been introduced: These are the “policy targets”, 
initially defined as national targets; they have been previously approved by the regional 
units and may be supplemented by regional targets. 

The plan is that they will lead to formally agreed targets at all levels of the labour ad- 
ministration in Germany, modelled on what is currently being piloted in a number of em- 
ployment offices. 



ERIC 



Siicli nnnlyscs iniglil be ven' useful nlso for coinpnnilive resen rcli into Inbonr-mnkel policy inslriimenls 
not included in the Eingliedcningsbilnn/.cn (licinscivcs (e.g. progrnniincs rinnnccd by llie Europenn 
SocinI Funds). 
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The overall objective is to control placement, counselling, and the use of labour market 
policy instruments by means of binding, quantitative goals related to labour market im- 
pact". The targets are defined in a complex process involving all organisational levels, 
then made binding; compliance is verified by the controlling system from time to time. 
Non-compliance triggers a dialogue intended to bring about behavioural changes. 

Figure 4 lists the business policy targets for .1999: cutting long-term unemployment; cut- 
ting youth unemployment; appropriate and immediate response to vacancies; prompt, 
customer-oriented allocation of unemployment benefits and assistance or support pay- 
ments for people in further training programmes. 

Sorting these target's in areas of action and indicators shows that a number of important 
data are at present unavailable and must be replaced with indicators as interim solutions. 
This generates (healthy) pressure on the information system to catch up to meet current 
needs . One example: although the flow into long-term unemployment may be calculated 
generally (Rudolph, 1998), it is not yet part of the statistics available. 

There are additional problems that arise from the fact that targets must be quantified'^. 
At the moment when realistic goals for certain selected parameters must be programmed 
(at the end of the previous year)' ' the key data for the budget and economic trend pro- 
jections will be available. But it is still difTicult, even at national level, to infer any such 
values for specific goals from the labour market trend projections. If historical data were 
available the systematic relationship between the specific parameters and the relevant key 
labour market data would have to be analysed, taking into consideration the impact of 
economic trends and structural changes ( shifts in trends, regional features ) as well as 
institutional changes. 

There were so many unsolved problems relating to a realistic definition of ’’anticipated 
indicator values” that the Bundesanstalt decided to make each regional level itself define 
1999 reference values for goals based on current trends. 

The emerging system of outcome goals may be an important addition to the interpreta- 
tion of the Eing/ieJoningshi/amen: 

Goal cotilrolling provides additional assessment criteria for the potentially very hetero- 
geneous and (sometimes) misleading results of the output evaluation. 

An employment office with a relatively low (gross) re-employment rate (for each type of 
measure) might nevertheless attain most of its goals (e.g. fewer inflows into long-term 
unemployment), because the employment office 



O 
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Olher menns of conlfol jjrc Icgnl regiilnlions and Ihe allocation of resources (available budget). 

One must also take note that such targets may not be manipulated. 

Indicator values to be expected in the existing situation which shall be affected as set out in the goals. 
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Samples of quantifications (federal level) 
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*) For some activities no appropriate statistical indicators are available yet, but these will be developed. 



- chose a mix of instruments best suited for the regional situation, 

in fact gave more attention to difficult cases, risking worse ’’gross re-employment 
figures”, while achieving a relatively high net re-employment effect'"’ (which is not 
reflected in the report) or 

because placement and counselling (as well as programmes ) were more intensively 
focussed on difficult cases. 

To some extent, the German labour administration could adopt the Austrian system, at 
least in the field of goal definition and controlling, because this system already has braved 
the introductory phase and created the requisite IT infrastructure. 

The Austrian goals include regionally adjusted projections of economic and labour mar- 
ket trends to an extent that is not yet available in Germany. Non-compliance with the 
goals triggers a search for the causes at all levels as well as any possible “behavioural 
changes” that might be necessary, e g. regarding the focus of activities (counselling, 
placement, measures) on the long-term unemployed. 

4.2 Quality management 

Goal controlling of this type aims exclusively at quantifiable targets - a general problem 
requiring additional activities. 

There is a pilot project entitled “Performance-related management in the Bundesanstalt 
fur Arbeit” which has been tested in four pilot employment offices since 1994 in co- 
operation with Bertelsmann Foundation (see Bundesanstalt fiir Arbeit, 1995 and 1996). 
Although it is true that the project looks at comparing performance, it also goes beyond 
this in looking for a new management concept which will increase employee motivation 
and optimise business processes with a view to greater customer satisfaction. Customers 
and employees are interviewed and trained in systematic goal-setting processes . 

Quantitative goals for the work to be done have been agreed on. However, they include 
a great number of partial indicators still awaiting systematic verification. There is also a 
consensus on qualitative goals: to improve operations and the organisation of work, the 
quality of customer service, leadership, co-operation and employee satisfaction. So far, 
however, there is no agreement on labour market impact. In the author’s view, this is a 
significant shortcoming. None of the quantitative indicators discussed so far sufficiently 
describes the quality of counselling, of vocational integration (duration, working condi- 
tions, employment to match qualifications), or the degree of involvement of and assis- 
tance to those individuals that are very hard to place. 

Quality management is also needed for the labour market outcome goals and must still be 
integrated into the emerging system of target agreements. This discussion has just 
started in the Bundesanstalt and solutions have yet to be found. The same will also apply 
to the emerging controlling concepts in other fields of public administration (e.g. .munici- 
palities). It is doubtful whether customer (or politician) satisfaction surveys provide suf- 

Disregarding polcnlial sccondaiy cITccls (e g. nuilliplicr c(Tccl), net effecls may never exceed gross 
effecls. However high gross elTocls may in some cases be based on veiy low not effects (e.g. re- 
employment of people who would luu'O been re-employed without the benefit of any measure). 
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ficient information on the quality of counselling and the enhancement of decision-making 
capabilities of those seeking help. In this respect, the experience of countries such as the 
Netherlands might help; they are focussing on quality management in their efforts to 
modernise the labour administration, going as far as to certify employment offices ac- 
•cording to the international quality standard ISO 9001 (Netherlands Employment Serv- 
ice, year n/a). 



5 Frame of refei ence: monitoring, conti olling, and evaluation 

This section offers a classification of the various individual trends described above. The 
literature on evaluation does not shed much light on evaluation or controlling: many de- 
finitions remain rather general and fuzzy. 

The Social Science Research Center Berlin collated information on similar activities in 
. EU member states to be included in a handbook on the evaluation of labour market po- 
licy that was published in 1966. In all of these countries these activities, whether referred 
to as controlling or (more frequently) monitoring, are being stepped up. 

What emerged from this is that, pragmatically speaking, monitoring is a tool 

<- to document performance or failures to reach preset goals (though not to explain them) 

- to open up and facilitate systematic feedback to the respective steering levels and 

- to be based on regularly conducted observation of statistical indicators such as: 

• input indicators (e g. funds), 

• output indicators (e.g. numbers of individuals successfully re-employed) and 

• outcome/impact indicators (e g. numbers of individuals added to the group of 
long-term unemployed) (Auer and Kruppe, 1996, 901). 

Thus, monitoring is a routine-based, quantifiable evaluation tool which needs to be com- 
plemented. Other tools for the same purpose might be non-quantifiable and non-routine 
information processing (and appropriate feedback to the decision-making level). Simi- 
larly, interpretative data might be gathered, i.e. complementary causal analysis research. 
This kind of data is needed to interpret the documented descriptive findings, e g. to di- 
stinguish between gross and net re-employment figures within the scope of the Emglie- 
cleriing.shilai izaii. 

In addition to this, and maybe even more importantly, research is required to select and 
interpret the indicators used for monitoring. What does a ratio of participants remaining 
unemployed recorded in the Linglicc/cningshilanz mean for the goal of “improving em- 
ployability”? What must be taken into account when the data only reflects the numbers 
of long-term unemployed rather than the inflow to long-term unemployment? 

Figure 5 is a diagram by the same authors designed as an ideal-typical model to define 
monitoring and to specify the elements that need to accompany it (either before or after): 
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Figure 5 



Monitoring Labour Market Policy 

(Auer/Kruppe 1996, p. 908) 




Notes 

1 Controlling the outflow oTbudgeis nnd the number of clients ser\'ed in relation to the goals set. 

2 Obseiwation of selected result indicators (c.g. postparticipation employment) 




1 AB Labour Market Research Topics 36 ( 1 999) 



26 



20 



- definition of goals at the appropriate administrative/political level, 

- selection of suitable indicators, 

- routine-based monitoring processes to collect and process the information yielded by 
the indicators, 

•- feedback loops for evaluation purposes and for policy-makers. 

Two things seem important here: 

1. Figure 5 clearly shows that both information on programmes (programme monito- 
ring) and labour market monitoring are required to obtain a comprehensive evaluati- 
on. 

2. This interpretation of monitoring includes'core elements of controlling: as has already 
been pointed out, the definitions of both remain rather fuzzy. Bearing in mind that 
such monitoring systems are restricted to routine-based, quantifiable and (usually) 
short-term effects (only short spells of employment, no consideration of the quality of 
the employment or whether it lasts), it is obvious that coiilrolUng must go' beyond 
monitoring ij it is to cover all the major goals (including goals defined as labour 
market goals). 

It is important to then ask what controlling is or what can it be? Here Postlep’s defi- 
nition (1994,11 ff) for local governments might be helpful. The following brief 
commentary on his five guiding principles with regard to current employment service 
trends might be a useful point of departure.'^ 

1. The enterprise is a self-steering .system (conti'ol looj)), i.e. feedback and correc- 
tion ensure goal-focussed action. 

Certainly, the concept of a control loop is fundamental and undisputed'^. Major 
constraints exist at the technological level especially, due to the current limits of 
statistics and IT systems. 

2. Such automatic control stres.ses .strategic (longer-term) goal setting/plamiing 
more than operative (short-term) goal. s/plans. This ensures the goal-focussed 
control of the overall organi.sation and at the .same time provides terms of refe- 
rence for the assessment of short-term activities. 

One of the consequences of stressing strategic, long-term planning for the em- 
ployment administration is a commitment to multi-annual targets. This has alrea- 
dy been done implicitly when business policy priorities were defined for 1999. In- 
dividual measures were decided according to the long-term problems on the la- 
bour markets, not according to short-term specific problems that might differ 




There is a corresponding derinilion in the BA's conirolling system which reads: 

“Conirolling intcgralcs various cicmcnis to cITcclivcIy and economically control the administration 
with the aim orachicving agreed goals. To this end. controlling must make the right amount of in- 
formation available at the right time and as comprehensively as required by the relevant decision- 
makers.” 

it likewise corresponds to recent concepts on the sociology of organisations for system-environment 
relationships. 
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considerably from one year to the next. 



In addition, these must be integrated into a commitment to higher-level goals set 

- by the federal government (provided for in Social Code III) 

- by the EU. 



3. 'I'he orgaiiisalional framework inii.sl allow for the decenlralisalion of decision- 
making powers relating to duties, funds (budgeting in the sense of financial 
planning), and human rescuirces, in tandem with lean management and consen- 
sus-based management. There should also be a performance-based, incentive 
programme. 

As mentioned previously, organisational changes are in the pipeline. However, 
this on-going process of change does not really facilitate the introduction of a 
new controlling system. 

Since 1998, Jsingliederungstitel legislation (i.e. a budget item with non-allocated 
funds) has essentially served the same purpose as “budgeting”. 

However, the constraints resulting from the legal provisions governing the civil 
service and its remuneration as well as budget law cannot be overlooked. 

4. A .'ieamlessly operating information network .should replace the current selective 
and needs-ba.sed .supply of information. This network would provide a data base 
for the process of defining goals, establish the framework to coordinate the ac- 
tion of individual parts of the orgaiil.sation, permit a verification of the effecti- 
veness of implemented measures and oj cost management, thus enabling the or- 
gani.sation it.self to function as a control loop. 

Certainly, horizontally and vertically integrated, seamlessly operating information 
networks are the core of any controlling system. Reference has already been ma- 
de to the iieed to link policy choices with the appropriate action and EingUede- 
rungsbilanzen, for example. When determining whether or not business policy 
goals were actually achieved, precisely how they were or were not achieved is in- 
itially immaterial. The Eingllederung.shilanz might be consulted as one primary 
indicator. This might also help prevent misleading interpretations. However, ad- 
ditional evaluation factors must be systematically included as well (e g. impact re- 
search results). 



5. Controlling requires broad .support for management across the entire range of 
deci.sions policy-makers must take. This means that the administration would 
need to supply extensive and sound data for policy deci.sions, explain the conse- 
quences of the various options and ensure the execution of the decisions made. 

Providing management support at all levels within the BA including at the level of 
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/the independent administration bodies and policy-makers outside the BA, sounds 
I laudable and would seem to go without saying. However, the important point is 
that complex control systems are only worthwhile if they are utilized, if they bring 
benefits and not just generate yet another set of figures. Costs and benefits must 
also be in proportion. Figure 6 shows the controlling modules that support a 
comprehensive evaluation system.”* 

• The purpose of goal controlling is primarily to help define performance and 
financial goals (steering goals) by showing the interdependence of various 
goals, goal/finance references and, by making available all goal-related infor- 
mation (in particular, the planned cost for different programmes and degrees 
of goal accomplishment). Possible discrepancies between the short and the 
long-term must be noted here, as well as the frequent lack of clear priorities 
at the level of higher-ranking political goals. One example: is it more im- 
portant to concentrate on prophylactic action (e g. long-term skills enhance- 
ment policy to improve the supply situation) or to focus on a hard core of 
long-term unemployed in a given situation, and how relevant might this be 
within the entire range ol stiategies pursued by employment promotion sche- 
mes? 

One must also consider how much can be achieved by the BA's activities, 
where other areas of policy-making must make a contribution, where are the 
current “mega-trends” on the labour market that are hard to affect in the 
short run: all of these are exciting themes for research and are very relevant 
for any BA controlling system. One example: how may the BA reduce long- 
term unemployment anyway, if long-term unemployment partly (to what ex- 
tent?) reflects problems of transition to retirement triggered by labour and so- 
cial law? 

• In the case of operative goal controllittg, the financial and human resources 
available annually must be taken into account. It must be determined what 
would happen to the different dimensions of these goals on the basis of sta- 
tus-quo assumptions - all the way down to the smallest regional units and the 
lowest decision-making levels. There must surely be considerable information 
gaps on how to set goals realistically. 

For example, how will the number of short-term filled vacancies change in the 
light of probable global and regional labour market trends, such as the num- 
bers entering long-term unemployment? It will be important to identify valid 
indicators and realistic expectations. In other words, though put in somewhat 
exaggerated terms: without a sound basis in this area, controlling may beco- 
me nothing more than a lottery, creating considerable frustration, even when 
falling short of the goals is not penalised and discussion and learning 
processes are generated (as they should be). 




See Postlep (1994,25) for local govemmenls. 
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systematic evaluation of instruments unsystematic 

(manipulation instruments, admistrative action) 
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development concepts goal attainment control 

priority goals based on an information network 



Informalion controlling is the overarching element linking the different fields 
together, yet at the same time it constitutes the biggest barrier to progress, 
because IT capabilities and ensuing bottlenecks in the statistics might obstruct 
many things that have already been recognized as being right. This has been 
true in the past, e.g. to get routine access to information on the employment 
future of ex-participants in job creation measures as had been available for 
those who participated in training and re-training programmes. As we have 
seen in the example of the Eingliedernngsbilanzen, legal provisions help to 
get things moving. 

Such substantial requirements from the perspective of controlling not only 
remain to be defined, but must also carry considerable weight in the debate. 
To put it bluntly: any information that is required for internal control purpo- 
ses only, and not for official statistics and research, ought to be definable 
within the controlling network. 

• Budget controlling, cost controlling and organisation controlling have not be- 
en discussed here at any length. These aspects are important but less relevant 
for labour market issues. However, the BA continues to work on these areas, 
as can be seen from the work being done on the “Arbeitsamt 2000” and 
“performance-based management”. There is nevertheless some crossover 
with labour market research, wherever impact research (e.g. of the new types 
of programmes) overlaps with “research into implementation” because this is 
the only way of determining whether certain goals were attained or not.'^ In 
the case of cost controlling, some sub-indicators might be necessary to permit 
a comprehensive cost-benefit analysis, which would be the most sophisticated 
and comprehensive assessment of individual programme types. 

Figure 6 also shows where impact research, which is mainly causal analysis, overlaps 
with controlling. Evaluation is the main focus here, and goal controlling is the common 
area of activity where monitoring, impact research and qualitative impact analyses from 
theory and practice meet - this is the only point where controlling in this area is comple- 
te. 



What is not shown in Figure 6, although Figure 5 on monitoring refects the same point 
and it has been mentioned repeatedly, is this: global and regional labour market analyses 
must also be included (goal size choices, development of valid indicators, determination 
of expectations, interpretation of results, complementing the findings relevant for the 
goals that cannot be captured by indicator-based systems.) 

6 Outlook: Trends in the El) 

At the EU level, the trend is towards “benchmarking” systems. Efforts are also being 
made to define common policy indicators to measure the achievement of the goals estab- 
lished by the employment policy guidelines adopted by the European Council in Luxem- 
bourg. 



See Dccke.Hiilscr.Wolfmgcr, (1997). 
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Benchmarking is “the search for best practices that lead to superior performance” 
(Camp, 1989; Tronti: 1998, 32), in other words basically benchmarking is a system es- 
tablishing outcome goals and defining indicators to measure the attainment of such goals. 
•To bring about positive changes, benchmarking analyses should always be seen in the 
same context as political action: learning and acting are mutually dependent and insepa- 
rable. Formal feedback loops show how close this is to controlling (Tronti: 1998, 33). 

The same problems regarding form and substance arise, e.g. for cross-sectional compari- 
sons of various regions. For this reason, the examples given by Tronti also show longitu- 
dinal comparisons within a region (the same argument was already discussed with regard 
to the Eingliederiingsbilanzen under Social Code 111). The resulting “ reference figure” 
provides a meaningful standard (Tonti: 1998, 37): 

“All indicators aie transformed, the highest performance (i.e. an unemployment level of 
zero, an employment rate of 100% for the working-age population, zero youth and long- 
term unemployment) always has the value 1 in the charts. These “benchmarks” are 
clearly unrealistic, but they simplify the standardisation process. Together, the seven val- 
ues form a calculable surface which is always comparable: if the single values are maxi- 
mised (e.g. if the unemployment rate is minimised), the surface figure will increase”. 

The EU-wide employment policy guideline indicator systems that are now under devel- 
opment (European Commission, 1998) also include labour market-related outcome in- 
dicators, such as flow into long-term unemployment, as well as measure-related input 
and output indicators (ratio of number of participants to the number of unemployed; em- 
ployment status of participants 6 or 12 months after participating in a programme ). 
These efforts might really change things and create a picture that can no longer be ig- 
nored by controlling and evaluating activities inside and outside the labour administra- 
tion. 
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